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ng Apparatus and Method 



t = o <-si 



initialize: p = 0, z = 0,p = 0 <- S2 



initialize parameters: for i = l...M;j = 1, K 
do S) = 0, a) = 1, b) = 1,4 = 0, a\ = 0, and a i = 1. 



observe s 0 , the system state at time 0 



S3 



S4 



S5 



The actor chooses an action ao based on its current parameters 0o and state so- 
The probability of choosing each possible action is given by: P(a 0 = a) = 7r0 o (so, a )- 



t = t + 1 



Implement the action a t 



S6 



S20 



From Figure 2. 



Move to the next state s t +i and observe the reward r t , S7 



Choose a new action a t +i based on 0 t and P(a>t+x = a) = 7r0 £ (s £ +i,a) <— S8 



Let f(a t) s t ) = P(a t = a|s t = 5), /(at + i,s t+1 ) = P(a t+ i = a|s t +i = 5). 
Evaluate partial derivatives of f{a u s t ) and /(a m , s t +i) with respect to 
all parameters comprising the vector 0 t . 



S9 



Go to Figure 2. 

Figure 1: Part I of the ACFRL algorithm 
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From Figure 1. 



Arrange the partial derivatives of f{a u s t ) and f(at+i,s t +i) 
into vectors X 1 and X l + X , each of length n. 






Compute Ql\{s u a t ) = Ewjtt/ and Q*(s m , a^i) = E? = iP% t+i 






Update p: p t +i = 


= Pt + a t (r t - p t ). 






Update p: p t +i =Pt + a t (r t - p% + Q£(*t+i, Q>t+i) ~ Ql\{s u <k))z t . 



<- S10 




YeS ^CFRL(l)?^ N0 



so or 



Yes 



No 



z £+1 = Az t + Vln7r^(5 t+ i 3 a t+1 ) <- S16 



= Zp + V In 7r^ (5 t+ i, a t+i ) 



Zt+i = Vln7T0 t (s t+ i,a t +i) <- S14 



S15 



S18 1 Update 0: fl w = fl t - i 9 t Qg;(^ ll fl t+1 )Vln7r gt (g t+1 ,a t+1 ) 1 <- S17 



Yes 



< ^For i ~ 1, n: (absolute value of rtfa component of 6 t+ i) > M z fy^^^ si9 



No 



Go to Figure 1 



Set absolute vai 



ue of ith 



component equal to Mj 



Figure 2: Part II of the ACFRL algorithm 
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Figure 3: Interference fuzzy labels used by the transmitters. 



0 




Backlog 



Figure 4: Backlog fuzzy labels used by the transmitters. 



